无人系统中离线强化学习的隐蔽数据投毒攻击方法

周雪; 苘大鹏; 许晨; 吕继光; 曾凡一; 高朝阳; 杨武

doi:10.11959/j.issn.1000-436x.2024264

您当前的位置：

首页 >

文章列表页 >

无人系统中离线强化学习的隐蔽数据投毒攻击方法

学术论文 | 更新时间：2025-01-14

- 无人系统中离线强化学习的隐蔽数据投毒攻击方法
- Stealthy data poisoning attack method on offline reinforcement learning in unmanned systems
- 通信学报 2024年45卷第12期页码：16-27
- 作者机构：
  
  哈尔滨工程大学计算机科学与技术学院，黑龙江哈尔滨 150000
- 作者简介：
  
  [ "周雪（1994- ），女，黑龙江牡丹江人，哈尔滨工程大学博士生，主要研究方向为网络安全、人工智能安全、强化学习。" ]
  [ "苘大鹏（1980- ），男，辽宁抚顺人，博士，哈尔滨工程大学教授、博士生导师，主要研究方向为网络流量安全监测、新型网络与人工智能安全。" ]
  [ "许晨（1996- ），男，山东菏泽人，博士，哈尔滨工程大学讲师、硕士生导师，主要研究方向为人工智能安全、自然语言处理、语音处理。" ]
  [ "吕继光（1987- ），男，黑龙江哈尔滨人，哈尔滨工程大学副教授、硕士生导师，主要研究方向为工业互联网应用安全、人工智能安全。" ]
  [ "曾凡一（1993- ），女，蒙古族，辽宁昌图人，哈尔滨工程大学博士生，主要研究方向为网络入侵检测、加密恶意流量分析。" ]
  [ "高朝阳（1999- ），男，河南驻马店人，哈尔滨工程大学硕士生，主要研究方向为网络安全、人工智能安全。" ]
  [ "杨武（1974- ），男，辽宁宽甸人，博士，哈尔滨工程大学教授、博士生导师，主要研究方向为网络与信息安全、人工智能应用及安全。" ]
- 基金信息：
  
  国家重点研发计划基金资助项目(2021YFB3101401);黑龙江省自然科学基金资助项目(TD2022F001);国家自然科学基金资助项目(U2003206;U20B2048;U21B2019;U22A2036;62272127);中央高校基本科研业务费专项资金资助项目(3072024XX0607)
- DOI：10.11959/j.issn.1000-436x.2024264
  中图分类号： TP393
- 收稿：2024-09-02，
  
  修回：2024-11-25，
  
  纸质出版：2024-12-25
- 稿件说明：
移动端阅览
周雪,苘大鹏,许晨等.无人系统中离线强化学习的隐蔽数据投毒攻击方法[J].通信学报,2024,45(12):16-27.

ZHOU Xue,MAN Dapeng,XU Chen,et al.Stealthy data poisoning attack method on offline reinforcement learning in unmanned systems[J].Journal on Communications,2024,45(12):16-27.
周雪,苘大鹏,许晨等.无人系统中离线强化学习的隐蔽数据投毒攻击方法[J].通信学报,2024,45(12):16-27. DOI： 10.11959/j.issn.1000-436x.2024264.

ZHOU Xue,MAN Dapeng,XU Chen,et al.Stealthy data poisoning attack method on offline reinforcement learning in unmanned systems[J].Journal on Communications,2024,45(12):16-27. DOI： 10.11959/j.issn.1000-436x.2024264.

摘要

针对现有离线强化学习数据投毒攻击方法有效性及隐蔽性不足的问题，提出一种关键时间步动态投毒攻击方法，通过对重要性较高的样本进行动态扰动，实现高效隐蔽的攻击效果。具体来说，通过理论分析发现时序差分误差对于模型学习过程具有重要影响，将其作为投毒目标选择的依据；进一步提出基于双目标优化的投毒方法，在最小化扰动幅度的同时，最大化攻击对模型性能产生的负面影响，为每个投毒样本生成最优扰动幅度。在多种任务及算法中的实验结果表明，所提攻击方法仅在投毒比例为整体数据1%的情况下，就能使智能体的平均性能下降84%，揭示了无人系统中离线强化学习模型的敏感性及脆弱性。

Abstract

Aiming at the limitations in effectiveness and stealth of existing offline reinforcement learning(RL) data poisoning attacks

a critical time-step dynamic poisoning attack was proposed

perturbing important samples to achieve efficient and covert attacks. Temporal difference errors

identified through theoretical analysis as crucial for model learning

were used to guide poisoning target selection. A bi-objective optimization approach was introduced to minimize perturbation magnitude while maximizing the negative impact on performance. Experimental results show that with only a 1% poisoning rate

the method reduces agent performance by 84%

revealing the sensitivity and vulnerability of offline RL models in unmanned systems.

关键词

Keywords

references

MNIH V , KAVUKCUOGLU K , SILVER D , et al . Human-level control through deep reinforcement learning [J ] . Nature , 2015 , 518 ( 7540 ): 529 - 533 .

SILVER D , HUANG A , MADDISON C J , et al . Mastering the game of go with deep neural networks and tree search [J ] . Nature , 2016 , 529 ( 7587 ): 484 - 489 .

苏新 , 孟蕾蕾 , 周一青 , 等 . 基于深度强化学习的海洋移动边缘计算卸载方法 [J ] . 通信学报 , 2022 , 43 ( 10 ): 133 - 145 .

SU X , MENG L L , ZHOU Y Q , et al . Maritime mobile edge computing offloading method based on deep reinforcement learning [J ] . Journal on Communications , 2022 , 43 ( 10 ): 133 - 145 .

ARULKUMARAN K , DEISENROTH M P , BRUNDAGE M , et al . Deep reinforcement learning: a brief survey [J ] . IEEE Signal Processing Magazine , 2017 , 34 ( 6 ): 26 - 38 .

LEVINE S , KUMAR A , TUCKER G , et al . Offline reinforcement learning: tutorial, review, and perspectives on open problems [J ] . arXiv Preprint , arXiv: 2005.01643 , 2020 .

DIEHL C , SIEVERNICH T S , KRÜGER M , et al . Uncertainty-aware model-based offline reinforcement learning for automated driving [J ] . IEEE Robotics and Automation Letters , 2023 , 8 ( 2 ): 1167 - 1174 .

CHEBOTAR Y , HAUSMAN K , LU Y , et al . Actionable models: unsupervised offline reinforcement learning of robotic skills [J ] . arXiv Preprint , arXiv: 2104.07749 , 2021 .

乌兰 , 刘全 , 黄志刚 , 等 . 离线强化学习研究综述 [J ] . 计算机学报 , 2024 : 1 - 35 .

WU L , LIU Q , HUANG Z G , et al . A survey of offline reinforcement learning research [J ] . Chinese Journal of Computers , 2024 : 1 - 35 .

SHI C , XIONG W , SHEN C , et al . Provably efficient offline reinforcement learning with perturbed data sources [J ] . arXiv Preprint , arXiv: 2306.08364 , 2023 .

KONYUSHKOVA K , ZOLNA K , AYTAR Y , et al . Semi-supervised reward learning for offline reinforcement learning [J ] . arXiv Preprint , arXiv: 2012.06899 , 2020 .

WU F , LI L Y , XU C J , et al . COPA: certifying robust policies for offline reinforcement learning against poisoning attacks [J ] . arXiv Preprint , arXiv: 2203.08398 , 2022 .

GOLDBLUM M , TSIPRAS D , XIE C L , et al . Dataset security for machine learning: data poisoning, backdoor attacks, and defenses [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023 , 45 ( 2 ): 1563 - 1580 .

WU Y , MCMAHAN J , ZHU X J , et al . Reward poisoning attacks on offline multi-agent reinforcement learning [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2023 , 37 ( 9 ): 10426 - 10434 .

刘艾杉 , 郭骏 , 李思民 , 等 . 面向深度强化学习的对抗攻防综述 [J ] . 计算机学报 , 2023 , 46 ( 8 ): 1553 - 1576 .

LIU A S , GUO J , LI S M , et al . A survey on adversarial attacks and defenses for deep reinforcement learning [J ] . Chinese Journal of Computers , 2023 , 46 ( 8 ): 1553 - 1576 .

ZHANG H , CHEN H , XIAO C , et al . Robust deep reinforcement learning against adversarial perturbations on state observations [J ] . Advances in Neural Information Processing Systems , 2020 , 33 : 21024 - 21037 .

YANG C L , KORTYLEWSKI A , XIE C H , et al . PatchAttack: a black-box texture-based attack with reinforcement learnings [J ] . arXiv Preprint , arXiv: 2004.05682 , 2020 .

ZHANG H , CHEN H G , BONING D , et al . Robust reinforcement learning on state observations with learned optimal adversary [J ] . arXiv Preprint , arXiv: 2101.08452 , 2021 .

SUN Y , ZHENG R , LIANG Y , et al . Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL [J ] . arXiv Preprint , arXiv: 2106.05087 , 2021 .

STANDEN M , KIM J , SZABO C . SoK: adversarial machine learning attacks and defences in multi-agent reinforcement learning [J ] . arXiv Preprint , arXiv: 2301.04299 , 2023 .

ZHANG X Z , MA Y Z , SINGLA A , et al . Adaptive reward-poisoning attacks against reinforcement learning [J ] . arXiv Preprint , arXiv: 2003.12613 , 2020 .

SUN Y , HUO D , HUANG F . Vulnerability-aware poisoning mechanism for online RL with unknown dynamics [J ] . arXiv Preprint , arXiv: 2009.00774 , 2020 .

FENG J , CAI Q Z , ZHOU Z H . Learning to confuse: generating training time adversarial data with auto-encoder [J ] . arXiv Preprint , arXiv: 1905.09027 , 2019 .

LIN Y C , HONG Z W , LIAO Y H , et al . Tactics of adversarial attack on deep reinforcement learning agents [C ] // Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization , 2017 : 3756 - 3762 .

KOS J , SONG D . Delving into adversarial attacks on deep policies [J ] . arXiv Preprint , arXiv: 1705.06452 , 2017 .

QU X H , SUN Z , ONG Y S , et al . Minimalistic attacks: how little it takes to fool deep reinforcement learning policies [J ] . IEEE Transactions on Cognitive and Developmental Systems , 2021 , 13 ( 4 ): 806 - 817 .

SUN J W , ZHANG T W , XIE X F , et al . Stealthy and efficient adversarial attacks against deep reinforcement learning [J ] . AAAI-20 Technical Tracks 4 , 2020 , 34 ( 4 ): 5883 - 5891 .

YU C M , CHEN M H , LIN H T . Learning key steps to attack deep reinforcement learning agents [J ] . Machine Learning , 2023 , 112 ( 5 ): 1499 - 1522 .

MA Y Z , ZHANG X Z , SUN W , et al . Policy poisoning in batch reinforcement learning and control [J ] . arXiv Preprint , arXiv: 1910 .05821 , 2019 .

RAKHSHA A , RADANOVIC G , DEVIDZE R , et al . Policy teaching via environment poisoning: training-time adversarial attacks against reinforcement learning [J ] . arXiv Preprint , arXiv: 2003.12909 , 2020 .

GONG C , YANG Z , BAI Y P , et al . Baffle: hiding backdoors in offline reinforcement learning datasets [C ] // Proceedings of the 2024 IEEE Symposium on Security and Privacy (SP) . Piscataway : IEEE Press , 2024 : 2086 - 2104 .

SUTTON R S , BARTO A G . Reinforcement learning: an introduction [M ] . 2nd ed . Cambridge : MIT Press , 2018 .

WATKINS C J C H , DAYAN P . Q-learning [J ] . Machine Learning , 1992 , 8 ( 3 ): 279 - 292 .

SCHAUL T . QUAN J, ANTONOGLOU I, et al. Prioritized experience replay[J ] . arXiv Preprint , arXiv: 1511.05952 , 2015 .

SZEGEDY C , ZAREMBA W , SUTSKEVER I , et al . Intriguing properties of neural networks [J ] . arXiv Preprint , arXiv: 1312.6199 , 2013 .

FOLEY H , FOWL L H , GOLDSTEIN T , ET AL . EXECUTE ORDER 66: TARGETED DATA POISONING FOR REINFORCEMENT LEARNING [J ] . arXiv Preprint , arXiv: 2201.00762 , 2022 .

SHIN D , DRAGAN A , BROWN D S . Benchmarks and algorithms for offline preference-based reward learning [J ] . arXiv Preprint , arXiv: 2301.01392 , 2023 .

LI A , MISRA D , KOLOBOV A , et al . Survival instinct in offline reinforcement learning [J ] . arXiv Preprint , arXiv: 2306.03286 , 2023 .

KRAFT D . Algorithm 733: TOMP–Fortran modules for optimal control calculations [J ] . ACM Transactions on Mathematical Software , 1994 , 20 ( 3 ): 262 - 281 .

FU J , KUMAR A , NACHUM O , et al . D4RL: datasets for deep data-driven reinforcement learning [J ] . arXiv Preprint , arXiv: 2004.07219 , 2020 .

TODOROV E , EREZ T , TASSA Y . MuJoCo: a physics engine for model-based control [C ] // Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems . Piscataway : IEEE Press , 2012 : 5026 - 5033 .

DOSOVITSKIY A , ROS G , CODEVILLA F , et al . CARLA: an open urban driving simulator [J ] . arXiv Preprint , arXiv: 1711.03938 , 2017 .

FUJIMOTO S , MEGER D , PRECUP D . Off-policy deep reinforcement learning without exploration [J ] . arXiv Preprint , arXiv: 1812.02900 , 2018 .

KUMAR A , FU J , TUCKER , G , et al . Stabilizing off-policy q-learning via bootstrapping error reduction [J ] . arXiv Preprint , arXiv: 1906.00949 , 2019 .

KUMAR A , ZHOU A , TUCKER G , et al . Conservative q-learning for offline reinforcement learning [J ] . Advances in Neural Information Processing Systems , 2020 , 33 : 1179 - 1191 .

SCHULMAN J , WOLSKI F , DHARIWAL P , et al . Proximal policy optimization algorithms [J ] . arXiv Preprint , arXiv: 1707.06347 , 2017 .

浏览量

923

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于数据特征的成员推理增强攻击方法

业务驱动的数据跨境风险机理研究

面向数据跨域安全流通的访问控制研究综述

基于DSMM的数据分类分级探索与实践

基于多级代理许可区块链的联邦边缘学习模型